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ABSTRACT j • 

There has been much literature on the effectiveness 
of program evaluation. Many feel evaluation has lost its 
justification ; others are more optimistic and feel evaluation can 
make a difference if the use levels are viewed _ realistically. 
Administrators must not regard program evaluation as merely complying 
with various agency requirements or representing a one-to-one 
matching between input and output where instrumentality is obvious. 
Because the evaluator' s credibility and positive relationship with 
his client are critical to the use of evaluative information , 
Meltsner (1976) developed a specific model for conceptualizing the 
evaluator f s role. The purpose of this study was to employ Meltsner' s 
model to explore administrators ' and evaluators ' perceptions of 
evaluation . Eighteen evaluators and 13 administrators from a large 
urban school system responded to 43 items which were grouped into 
clusters through factor analysis. Findings indicated the mental 
paradigms of evaluators and administrators were very different . ^More 
research is needed in other settings to validate findings of this 
study\. To facilitate evaluation use , exploration of interpersonal 
factors is recommended. (HFG) 
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The literature on the use of program evaluation information contains numerous 
desultory comments regarding the use of evalative information in local school 
systems. For example, Rippey (1973, p. 9) concluded that "at the moment, there seams 
to be no evidence that evaluation, although the law of the land^ contributes anything 
to educational practice other than headaches for the researcher, threats for the 
innovators, and depressing articles for journals devoted to evaluation." 
Practitioners in local education agency (LEA) settings offer similar views. For 
example, Hoi ley (1979, p. 2) notes that "in an ideal world we wouldn't have to worry 
about utilization. Educators would be eagerly awaiting our findings and would 
promptly rush to put them into practice. I don't have to tell you that isn't- 
happening." These concerns are serious, because "if evaluative information is not 
useful, if it does not meet the needs of decision makers, then evaluation has lost 
its justification" (Alkin, Kosecof f , Fitz-Gibbon, & Seligman, 1974, p. 1) . 

Fortunately, more recent studies suggest that these pessimistic views may be 
somewhat unrealistic. Thus Alkin, Daillak, and White (1979, p. 16) ■> enpfcasis in 
original) argue that "taken together, the studies and our observations* and 
experiences suggest to us that evaluation can make a difference, that it does so more 
often than the published critiques suggest, that sane school districts 
characteristically produce a high proportion of useful evaluations, and that seme 
evaluators have acquired skills that allow them to carry out technically competent 
and pr ogr amna t i cal ly influential evaluations." In an extensive review of the 
literature on the use of evaluation in LEA settings, King, Thompson, and Pechnan 
(1981) identify at least two factors that may have led evaluators to became 
unreal istically disheartened about use levels. First, although evaluators may always 
intend that results should be used, administrators may tolerate some evaluation 
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endeavors merely to comply with various agency requirements. Alkin (1980, p; 3) 
makes the point using the analogy of a garden party: "Suppose the host... should 
insist that each of the guests periodically rate the quality of the party, or the 
drinks or the food, etc.— it can't really be expected to have much impact. This 
somewhat peculiar, externally imposed requirement will be tolerated as part of the 
'price of admission,' so to speak, but it won' t really change the behavior of 
individuals." 

A second and even more important perspective on the use of evaluative 
information involves the difficulty of detecting use. As Wise.: (1980, p.. 24) 
suggests, "if there is an evaluation utilization problem, it is not that 
decision-makers do not use the information they receive, it is that evaluators cannot 
easily see their information being used Hn the incremental ism of real-world 
decision-making." This has become more apparent as evaluators have recognized that 
there are several types of use that vary in their visibility and impacts. 
Traditionally, many evaluators unconsciously looked for instrumental use of 
evaluative results, i.e., the making of go/no go decisions that purportedly could be 
detected by tracing "a single decision to a single piece of knowledge in a one-to-one 
matching between input and output" (Pelz, 1978, p. 354). This type of use rarely 
occurs, though examples of instrumental use certainly can be identified (e.g., Alkin, 
Daillak 5 White, 1979, p. 224). 



However, probably the more common form of Use has been termed conceptual 
i.e., use "influencing a policymaker's thinking about an issue without putting 
information to any specific documentable use" (Rich, 1977, p. 200) . As Cronbach, 
Ambron, Dornbusch, Hess, Hornik, Phillips, Walker, and Weiner (1980, p. 193) 
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correctly note, "stimulating a discussion that leads to gradual change in prevailing 
views is very likely the most important effect of evaluation research." In short, 
because evaluation use can be difficult to observe, seme evaluators may tend to 
underestimate evaluation impacts. Such a view does help resolve the seemingly 
paradoxical finding (e.g., Alkin et al. , 1974; king & Thompson, 1983) that 
administrators themselves do consistently report that evaluation is useful. 

Another consistent finding in the literature is that the e valuator's credibility 
with clients, in all its forms, is critical to the use of evaluative information. 
Thus Holley (1979, p. 8) reports that "utilization is usually the result of the 
relationship between the evaluator and the user more, than anything else. If the user 
knows and respects the evaluator, utilization has its highest potential." Similarly, 
based on a review of empirical literature, LevitbiTand Hughes (1979, p. 21) indicate 
that "the evidence on dissemination suggests that informal communication that cuts 
the red tape may enhance utilization, although quality of information may sometimes 
suffer and dissemination will be haphazard." Thompson and King (1981) have emphasized 
the importance of reliance upon theoretical propositions to explain why use Phenomena 
occur. In light of House's (1977, p. 5) suggestion that "evaluations themselves , I 
would contend, can be no more than acts of persuasion," communication theory has been 
employed to explain these use dynamics . The theory' s origins can be traced back ' to 
the Rhetoric of Aristotle (7/1886, p. 11) , who himself argued that "we may 
practically lay it down as a general rule that there is' no proof so effective as that 
of the character of tne speaker, i.e., ethos ." Communication theory provides fairly 
specific guidance for evaluation practice (Thompson, 1981) , and seemingly supports 
the recommendation * of the Joint Committee on Standards for Educational Evaluation 
(1981, p. 24) that "if the confidence and trust of these client audiences cannot be 
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secured, the evaluator should seriously consider not proceeding." 

Meltsner (1976) has offered a specific model for conceptualizing the evaluator 's 
role. Meltsner suggested that evaluator s can be differentiated from each other by 
their reliance (high or low) on combinations of analytical and political or 
interpersonal skills. The four unique combinations of degrees of reliance on these 
two skills define four "types" of evaluators. Though seme practicing evaluators 
regard at least Melstrier's labels for the four "types" of evaluators as being 
pejorative (Thompson, 1982, pp. 15-17) , empirical research suggests that evaluation 
clients are sensitive to manifestations of these two skill dimensions (Thompson & 
King, 1981). Thompson (1980, p. 64) found that two -* types" of evaluators," i.e., 
those that place high and low reliance on analytical and political skills, 
respectively, and those that place low and high reliance on analytical and political 
skills, respectively, may not be common in LEA evaluation settings. 

The purpose of the present study was to employ Meltsner 's model to explore 
administrators' and evaluators' perceptions of evaluation. Direct comparisons of 
these two perspectives appear to be non-existent in the current literature. More 
specifically, the study employed Meltsner 's framework as a basis for addressing two 
research questions. First, do LEA administrators and LEA evaluators have similar 
views of evaluation and the evaluator 's role in the educational process? Second, 
given that clusters of study participants can be identified, which attitudes and 
opinions regarding evaluation differentiate the clusters? 

Method 

Participants 



G 

ERIC 



Page 5 

The sample consisted of 18 ^valuators and 13 : upper-level administrators employed 
in a large, urban, public school system. The sample of evaiuators consisted of 
full-time, professional members of the system's evaluation department. The 
administrators who participated in the study included six persons whose 
responsibilities primarily included administration of curriculum matters. An 
additional seven administrators had line administrative responsibilities. 

Instrumentation 

Meltsner (1976, p. 48) has suggested that various types of evaiuators can be 
differentiated on the basis of factors such as central professional motivation, 
standards of success, credibility vested in various forms of prooof , and orientation 
toward evaluation projects involving shorter or longer time perspectives. Thirty-six 
itens were developed to measure these variables • Typical items were: "Good 
evaiuators judge their success by what other evaiuators think of their work*" and 
"Research skill is only one of the tools which can be used in evaluation studies." 

More recent literature also recognizes the importance of evaiuators actively 
working to see that evaluation results are used. For example, Polivka and St eg 
(1978/ p. 697) argue that "traditionally, the evaluator has been very hesitant to 
claim any responsibility for the" use of his/her findings. This approach has helped 
make it very easy to ignore evaluation results." Similarly, the members of the Joint 
Canmittee on Standards for Educational Evaluation (1981, p. 47) suggested that 
" evaiuators must not assume that improvements will occur .automatically once the 
evaluation report is completed. Such improvements must be stimulated and guided, and 
evaiuators can and should perform an important role in this process." Consequently, 
an additional seven items were employed to measure judgment of evaiuators 1 efforts to 
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optimize the use of evaluative information. 
Procedure 

The 31 subjects individually sorted the 43 statements into a quasi-normal 
distribution, in the usual' fashion of a Q-study, based upon extent of agreement with 
erch of the statements. A* Kerlinger (1973, p. 598) notes, "one can rarely 
generalize to populations frcm Q person samples. Indeed, one usually does not wish 
to do so. Rather, one tests theories on small sets of individuals carefully chosen 
for their •known 1 or presumed possession of some significant characteristic or 
characteristics here job role ... Used thus, Q is an important and unique approach 
to the study of psychological, sociological, arid educational phenomena." 



Results 

Q-technique factor analysis was employed to address the study's two research 
questions. The technique groups subjects into clusters based on ^similarities in 
response patterns. However, unlike the more commonly applied R- technique methods, 
Q-technique analysis can be utilized only when the number of subjects is small 
relative to the number of variables in the study. Thus a preliminary. Q-technique 
factor analysis was conducted to exclude 11 of the study's 31 participants. This 
represents operalization of the conservative convention that the number of subjects 
in a Q-analysis should be at least one less than one-half the niinber of sorted 
statements. . j 

Five subjects for each of the four factors identified in the preliminary 
analysis were selected for inclusion in the final analysis; these 20 subjects were 
identified by selecting the five subjects per factor who had the highest correlation 
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with each of the factors identified in the preliminary analysis. This procedure was 
hot considered to bias subsequent results with respect to the study's two research 
questions ^ since the subjects' job roles (administrator vs. e valuator) and statement 
preferences were not consulted. Instead, the procedure served to provide a final 
solution with "simpler structure 11 since the 11 subjects who were excluded by 
definition contributed less to the formation of the clusters identified in the 
preliminary analysis. This theoretically should provide a more stable framework for 
addressing the study's two research questions. 

The clusters of the regaining 11 evaluators and nine administrators are 
presented in Table 1. All principal components with eigenvalues greater than one 
were extracted. The factor scores for the 43 statements that differentiated the four 
factors are presented in Table 2. 

INSERT TABLES 1 AND 2 ABOUT HERE. 



The study's first research question asked whether administrators and evaluators 
have similar views of evaluation and the evaiuator's role. This question can be 
addressed by interpreting the results presented in Table I. It is noteworthy that 
each of the four clusters of subjects consists of both administrators and evaluators. 
The first two factors, which accounted for more variance than did the second two 
factors, in particular consisted of a mix of evaluators and administrators. Even the 
last two factors included several evaluators who also were associated with one of the 
first two factors and thus had response patterns that were associated with those of 
seme administrators. 
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The study's second research question asked which attitudes and opinions define 
and differentiate the identified clusters of subjects. This question can be 
addressed by consulting the factor scores for the 43 statements presented in Table 2 
for each of the "four factors. With respect to the first factor, although the Table 1 
results indicate that four of the evaluators were associated with the factor, it is 
noteworthy that ail of the administrators involved in the final analysis were 
associated to an appreciable degree with this cluster of subjects. The Table 2 
values suggest that these subjects believe that evaluators must emphasize long-term 
impacts when evaluating programs and believe that good administrators at least try to 
use evaluative information when making decisions. The factor scores for items seven 
and 34 suggest that the subjects believe that evaluators should be particularly 
sensitive to considering program impacts on program participants. The factor score 
for statement 42 suggests that these subjects do not view evaluation as an inherently 



political process. This cluster of subjects seesns to view tshe evaluator as a patient 
and service-oriented agent woking on behalf of program clients,; 

The second factor also consisted of many of the administrators and several of 
the evaluators. The Table 2 results indicate that these subjects believe that 
evaluators should not concern themselves with judging the worth of program goals. 
These subjects believe that evaluators should avoid making policy recommendations. 
The subjects also believe that non-quantitative ways of evaluating programs are 
valid. Scores for several items, e.g., items nine, 32, and 40, suggest that these 
subjects believe that evaluators should interact, regularly with administrators. This 
cluster of subjects seems to view the evaluator as a responsible agent of 
administrators and as a professional who is fairly .legalistic and concrete in 
defining job description. 
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fts the Table I results indicate^ the third factor essentially consisted of five 
or six evaluators and one adninistrator . The factor scores for the 43 statanents 
with respect to this factor indicate that the subjects resent evaluations being 
conducted merely to justify a decision that has already been taken. The subjects 
believe that it is acceptable that some evaluations generate controversy. The 
subjects believe that evaluators should solicit administrative input regarding the 
focus of an evaluation. The subjects do not believe that evaluators should evaluate 
the philosophies underlying programs. The subjects believe that evaluators should 
use various forms of proof, shcpld consider the timing of their efforts ^ and should 
try to get all available information before adting. This cluster of subjects seems 
to view the evaluator as a person who should be practical and politically sensitive 
in making decisions , e.g., selecting forms of proof and timing the release of 
reports. v 

The fourth factor essentially oonsisted of fiye evaluators and one 
acininistrator. The subjects who clustered togettier on this factor believe that 
evaluators should make policy recommendations to aAninistrators, as indicated for 
example by factor scores for statements 17 and 22. The subjects believe that 
evaluators should consider both short- and long-term issues when evaluating programs. 
It is considered acceptable that 4 sane evaluations provoke controversy. Good 
evaluators are assumed to be good with both people and numbers. This cluster of 
subjects seons to view the evaluator as an activist professional whose purpose is to 
improve the operation of institutions by adopting balanced perspectives regarding 
program ends and the means selected to achieve these ends. 
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Several general patterns emerge from these various results. First, the finding 
that each factor included both administrators and evaluators suggests that at least 
some representatives of each group hold views of evaluation that are compatible with 
the views of some counterparts. Second, the finding that most of the administrators 
clustered together on the first two factors suggests that aSriinistrators; views of 
evaluation and of evaluators are more homogeneous than are the corresponding views of 
evaluators. / 

The ^f actor scores for statement 18, presented in Table 2, are worthy of canment. 
The result suggests thst the first two clusters of study participants, which included 
the administrators, believe that evaluation is influential. The scores for the itan 
on the third and fourth factors, which consisted primarily of evaluators, indicate 
that these two clusters of subjects are less optimistic about evaluation's impacts. 
This result replicates previously mentiond findings already in the literature, it is 
conceivable that evaluators underestimate the impacts of evaluation while 
administrators overestimate evaluation's current utility. Hopefully this paradox 
will be more directly investigated in future research that examines particular types 
of utility. . Granville (1977) , for example, investigated which types of decisions 
might be best facilitated by LEA evaluation , but the study did not compare 
administrators' and evaluators' views regarding specific types of decisions and the 
corresponding utility of evaluation. v \ 

Overall, the results provide some support for a conclusion that the "Two 
Communities" concept (Caplan, Morrison & Stambaugh, 1975) may characterize LEA 
settings. The concept suggests that the perspectives or mental paradigms that 
administrators and evaluators employ to guide behavior and interpret events are very 
different and that consequently communication between the two "communities" is 
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impeded. The empirical finding (Lyon, Doscher, McSranahan & Williams, 1978) that 
many LEA evaluators and'LEA administrators have different backgrounds certainly lends 
credence to this possibility. The administrators in the present study seemed to be 
more reflective about the end purposes of evaluation than were at least seme of the 
evaluators. Of course, it is heartening that a majority of the subjects in the study 
were associated with the first factor, and thus apparently place considerable 
emphasis on evaluation as a vehicle to improve program service to program clients. 

However, the conclusions reported here must be generalized with caution, since 
this research involved subjects from only one school system. Thus additional 
research in other LEA settings is required. The literature contains very few 
theoretically grounded, direct comparisons of LEA administrators' and LEA evaluators' 
views of evaluation and the evaluation process. This situation is distressing 
because, as Gurel (1975, pp. 27-28) has observed, "the major barriers to successful 
evaluation are not technical and methodological, though these are certainly important 
and worthy of further effort, but are rather the structural constraints and 
requirements and the interpersonal relationships which characterize the evaluation 
endeavor." The development of theoretical models and the completion of studies 
involving these interpersonal dynamics will not be easy. Such efforts may also 
offend the sensibilities of some persons Who regard evaluators as individual people 
whose behaviors and biliefs are shaped by idiosyncratic factors that make the 
dynamics purportedly both impossible and inappropriate to study. But our ability to 
make educational evaluation in LEA settings reach its full potential may ultimately 
require that we moderate our sensitivities and continue to explore the interpersonal 
factors that really seem to make evaluation use' happen. The alternative is to retain 
limited understanding of and control over the impacts of the profession. 
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Table 1 

Varimax Rotated Factor Structure 
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Note: Structure coefficients greater than an absolute value of .4 are indicated with 
an asterisk. 
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Table 2 
Factor Scores on Items 

IV Item 

•2.1 1. Good evaiuators evaluate programs primarily in terms of 

th^ir long- term im^cts . 
.1 2. Most quality evaluation reports place a heavy emphasis 

on research results . 
■1.9 3. The evaiuator* s job is to evaluate programs or 

policies; it is not the evaiuator 1 s job to judge the 

purposes of programs. 
-.7 4. It is more important for evaiuators to know a little 

about a lot of things rather than to know a whole lot 

about a few. things; 
.8 5. Good adninistrators actively try to use evaluation 

results in making decisions.- 

1.1 6. Good evaiuators consider the practical aspects of how 

programs work when they conduct evaluation studies. 
.7 7. Projects should be evaluated primarily in terms of how 

well they. help their clients. 
.3 8. Good evaiuators hate making recanmendations that 

probably will not be accepted. 
-.1 9. Good administrators are usually too busy to meet very 

often with evaiuators . 

-.9 10. Good evaiuators want to feel influential and important. 
1.0 11. Research methods are important in evaluation, but so 

are some other forms of insight that should be used 

along with than. [_ 

.9 12. Evaiuators can judge their success by how much access 

they have to the people who make ail the decisions. 
.0 13. Good^ evaiuators strongly resent it when they are asked 

to perform an analysis in order to justify a decision 

which has already been made. 
-.6 14. Good administrators realize that evaluation results are 

very limited in their ability to mirror program 

realities, therefore good administrators place limited 

emphasis on evaluation evidence when making decisions. 
.0 15. Visiting projects and observing activities in an open, 

non-directed way can be an important part of a good 

evaluation. 

-.9 16. Good evaiuators strongly desire to be told that they 

are doing a good job. 
2.0 17. Good evaiuators _ may know what pol icy or program 

decisions should be made, but it is, not their job to 

make policy recommendations. 

1.2 18. To be honest, evaluation probably \ doesn't influence 

decision making very much. 
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37. 



It Is very important to good evaiuators to work oh 

projects that are "where the action is." 

Good evaiuators frequently enjoy doing studies, that 

meet the short-run needs of decision makers , but they 

also enjoy working on analyses with longer-range time 

perspectives^ z _ _• 

If an evaluation result is not immediately used for 

decision making purposes, tne evaluation must have been 

bad.. 

The job of a good evaluator is to give decision makers 
both information and recommendations . 

Most good, evaiuators prefer working on problems that 

can be solved fairly quickly. \ 

Good evaiuators think a lot about "timing" when they 

decide when to release their evaluation reports. 

Good evaiuators enjoy working with people. \' 

Good evaiuators actively work to get administrators to 

use evaluation results as part of decision making. 

It is important for good evaiuators to acquire even 

skills which they aren't certain they will ever use. 

Good evaiuators are successful mostly because of their 
communication and management skills. 
Good evaiuators are good with both people and numbers. 
Good evaiuators judge their success by what other 
evaiuators think of their work. 

If a good evaluator disagreed with the philosophy of a 
program, the evaluator ought to argue against the 
philosophy in discrete discussion with the right 
decision makers * 

Good evaiuators realize that adninistrators can 1 t 

reasonably place very much emphasis on evaluation 
information, so good evaiuators don 1 t try to "twist 
adninistrators 1 arms" to get them to use evaluation 
results in making decisions. 

Good evaiuators would someday like to be decision 
makers rather than evaiuators. 

Good evaiuators don't mind sticking their necks out in 
an analysis, if jbhey think it will make things better 
for the people the organization is supposed to be 
serving. \ 

Good evaiuators like to get credit for their work, but 
it's not absolutely essential to them that they get 

credit. ■'_<_* ... 

Good evaiuators feel comfortable selecting the focus ' 
.for an evaluation report, because they know their 
bosses' preferences. 

To sane extent, the good evaluator believes it is the 
evaluator 's , job to please the people they ■ 
" psychologically" work for, even if these people aren't 
their official bosses on the organizational charts. 
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38. 


Good evaluators don't feel they have to know every 
single conceivable fact before they write an evaluation 
report. 
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39; 


An evaluation analysis is poorly done if it creates 
more furor than it resolves . •' 
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4b. 


Good administrators interact often with program 
evaluators; 
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41. 


Research skill is only one of the tools which can be 
used in evaluation studies . 
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42. 


Evaluation would be more productive if everybody simply 
admitted that it is ah inherently political process . 


.0 


.3 


.2 


1.1 


43. 


Good evaluators strongly desire to work in ah 
environment with other bright and knowledgeable 
researchers . 



Note ; Factor scores were standardized such that the mean on each factor was zero and 
the SD on each factor was one. Positive scores indicate that the subjects who 
clustered together to constitute a factor tended to agree with statements; 
negative scores indicate more disagreement with statements. 
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